1. Field of the Invention
The present invention relates to data integration applications and, more specifically, to the use of a common type-based data abstraction layer when building data integration applications.
2. Discussion of Related Art
When a database system is upgraded or replaced, the existing data must be transferred to the new system. This process, called data migration, is traditionally expensive as database systems become larger, more diverse, and more complex. Planning and executing a data migration application consumes valuable resources and can often result in considerable downtime. Also, mistakes in data migration can lead to data corruption, which is not an acceptable risk for institutions that handle sensitive data.
These difficulties are compounded when it is necessary to combine and transform data from several different data storage systems, a process known as data integration. Data integration applications must reconcile data from several potentially incompatible storage systems, convert these data into a unified format, and load the new data into the target storage system. These are complicated tasks, and they require careful planning and detailed knowledge of the structure of the source and target databases. Errors in data integration are common, difficult to diagnose, and expensive to fix.
Traditional approaches to data integration suffer from several problems related to the business logic and mappings that are embedded in the applications. In order to effectively enforce business policies and ensure that the applications conform to an organization's standards, data integration developers must hard-code business logic, or rules, in the data integration applications to implement the standards. Embedded business logic is problematic in that the traditional business users or data governance experts responsible for the standards are wholly dependent on more technical users to implement the rules in the data applications.
This approach is also problematic because it is difficult for an organization's stakeholders to have visibility into the embedded logic of the applications as needed to either support the process of developing the applications but, more importantly, to support audit requirements. This is because such business logic is implemented in some form of transformation module or language that is not defined or managed as a declarative and reportable metadata structure of the application.
Further, implementation of the data rules as embedded logic makes it difficult for the applications to be easily reused with different source and target data systems. This is because the rules are usually performing some form of type conversion, validations, and transformations specific to the sources and targets integrated by the original application. While it is possible to reuse those parts of the application that do not implement such rules, it is not possible to do this without creating a new modified copy of the data integration application. Also, applications created using conventional data integration tools do not reliably separate the source- and target-independent logic from the logic that is source- and target-specific, making it difficult to isolate the portions of the application that can be reused without modification.
In light of these problems, there exists a need for an improved method of developing database applications that minimizes the costs and risks associated with data migration and data integration.